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EDUCATIONAL PROGRAM AUDITING: THE RESEARCH PERSPECTIVE 

Richard Seligman 

Center for the Study of Evaluation 
University of California, Los Angeles* 

This paper presents selected results of a major study on the topic of 
evaluation and decision making which was conducted at the Center for the 
Study of Evaluation (Alkin et al. , 1974). The purpose of this paper is to 
shed additional light on what we know about independent educational accom- 
plishment audits and the manner in which they have been carried out. While 
much has been said about the notion of educational accomplishment auditing, 
there has been no attempt until now to systematically gather empirical 
evidence on this procedure. 

As part of the larger study cited above, data on the audits was obtained 
for a group of thirty-nine bilingual education projects funded under Title VII 
of E.S.E.A. The audit reports covered the bilingual programs during the 1970- 
71 school year. These reports were analyzed by at least two independent 
reviewers. The reviewers' ratings were recorded on the Audit Data Sheet, an 
instrument designed to tap essential information on the manner in which the 
audit had been conducted. In those cases where the raters' judgments did not 
coincide, a third rater reviewed the audit report in question and served as 
an independent adjudicator. This procedure yielded interesting and useful 
data on three principal features of the educational program audit: the 
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procedures employed by the auditors, their judgments on the quality of the 
evaluation procedures, and the scope of the audit activities. 

Audit Procedures 

On the Audit Data Sheet, 52 audit procedures v/ere enumerated. These ■ 
were derived from a careful examination of the literature on independent 
audits, including USOE guidelines ( Programs under Bilinguf^l Education Act , 
Manual for Project Applicants and Grantees, HEW, 1971) and the Handbook for 
Educational Program Audit (Morin, 1971). Table 1 presents the audit pro- 
cedures which were employed by at least two-thirds of the 39 auditors. These 
data were obtained in response to the item "Indicate which sources of data 
the auditor UGC^d as a basis for his report ..." 

As might be expected, one of the two most commonly used data sources 
was the site visit. It is worth noting, however, that two projects in the 
sample did not have an on-site visit from the auditor. The other most conmon 
" technique, used by 88% of the auditors, was the review of tests which had 
been administered by the evaluator. In most cases, however, the auditors 
did not indicate what criteria they used in judging the adequacy or appro- 
priateness of the tests. Furthermore, it was rare for the auditor to question 
or challenge the evaluator's selection of instruments. The third most frequent 
data source was the final evaluation report with 36 of the auditors using these 
reports in conducting these audits • The fourth technique was examination of 
the raw data gathered by the evaluator; such data included test and question- 
naire results. While 36 of the auditors used this source of information, 
unfortunately most of the audit reports did not go on to indicate how these 
data were examined by the auditor and what procedures, if any, were used to 



Table 1 

Data Sources Employed by at Least 
Two-Thirds of the Auditors in the Sample (N=39) 



Data Source 


N 


% 


Site visit 


37 


88 


Evaluator-administered tests 


37 


88 


Final evaluation report 


36 


86 


Ravi data gathered by evaluator 


33 


79 


Interviews 


29 


69 



Table 2 

Data Sources Employed by Fewer than One-Third 
of the Auditors in the Sample (N=39) 



Data Source N % 



Auditor-administered tests 
or questionnaires 


1 


2 


Correspondence with staff 


2 


5 


Project financial records 


3 


7 


Evaluation contract 


5 


12 


Evaluation proposal 


10 


24 


Continuation proposal 


9 


21 


Interim continuation report 


7 


17 


Archival information 


12 


29 
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verify the data gathered by the evaluator. There were relatively few 
instances where the accuracy of the evaluator's data was questioned by the 
auditor. Many of the auditors also reported conducting interviews with 
project personnel, generally with the project director and other staff; 
interestingly enough, only seven audits reported interviews with students. 

In contrast to the most commonly employed audit techniques, one might 
also consider some of the least frequently employed audit procedures. Table 
2 shows which audit procedures from among the 52 listed were employed by 
fewer than one-third of the auditors. Only one of the auditors administered 
his own tests or questionnaires. Few auditors reported looking at "archival" 
information such as newspaper clippings, staff correspondence, students' 
cumulative records and letters from members of the community. Most of the 
auditors did not report the use of either the evaluation proposal or the 
interim evaluation report in their work. This last finding may be related 
to the fact that the data presented here are based on the contents of the 
final audit report; one would expect that several of these items might have 
been treated in the interim audit reports. 

A consideration of the procedures which were used by the auditors leads 
to the conclusion that four commonly acknowledged audit practices v/ere 
employed: site visits, reviews of instrumentation, examination of final 
evaluation reports, and interviews with staff. The auditors did not admin- 
ister their own tests, nor did they make extensive use of project archival 
data. 

When reading about educational accomplishment auditing, one invariably 
comes across the suggestion that part of the audit function should include a 
re-analysis of some, if not all, of the data which the evaluator has collected 
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and reported. Only in this manner, it is contended, can the auditor truly 
verify the evaluator's findings. The study yielded few, if any, examples of 
the re-analysis of evaluators' data by the -auditors. Nevertheless, it seems 
fair to conclude that, in general, auditors were doing those things which 
auditors are supposed to do. The data suggest, however, that auditors should 
be more attentive to describing in detail the procedures which they employ, 
A number of the auditors were rather vague about just what they did; there 
is no room for ambiguity in these matters if the findings of the auditor are 
to be accepted with confidence. 

Auditor's Judgments on the Quality of Evaluation Procedures 

One of the primary functions of the independent educational accomplish- 
ment audit is to judge the quality of the evaluation. A major section of the 
Audit Data Sheet dealt, therefore, with those aspects of the evaluation about 
which the auditor commented and made qualitative judgments. The raters 
assigned a score ranging from -3 to +3 to elements of the evaluation. At 
one extreme, -3, the auditor was judged to have said that the evaluation was 
poor with respect to that particular item. At the other extreme, +3, the 
auditor was thought to have made a most favorable judgment about the evalua- 
tion. In the middle, 0, the auditor was judged simply to have said that an 
item had been attended to by the evaluation but made no attempt to assess 
the quality of the evaluation effort. Table 3 lists some of the components 
of the evaluation which were typically commented upon by the auditors. 

The item "accuracy of data gathered by the evaluator" received a mean 
rating of 1.75, that is, above average in quality. Thus, while the auditors 
did not re-analyze the evaluator's data, neither did they question the 



Table 3 



Mean Ratings of Auditors' Judgment of the 
Quality of Selected Components of the Evaluation (N=39) 



Evaluation Component Mean Rating* %>1 %>0^ 



General evaluation design 


1.22 


36 


77.88 


86.11 


Assessment techniques 


.94 


36 


63.89 


86.11 


Testing iFv^slTuments 


.77 


35 


65.71 


82.86 


Interpretation of results of analysis 


o32 


34 


50.00 


58.82 


Analysis of raw data 


.30 


33 


48.48 


60.61 


Appropriateness of data collection 
techniques 


.97 


32 


68.75 


81.25 


Format of evaluation report 


.44 


32 


50.00 


59.38 


Accuracy of gathered data 


1,75 


28 


75.00 


85.71 


Completeness of report 


0.00 


28 


42.46 


46.43 



*0n a scale ranging -3 = auditor judged evaluation poor on the component 
to +3 = auditor judged evaluation to have been done quite well on the 
component, 

^Number of audit reports in which judgments were made. 

'^Percentage of responses greater than +1.0 on a scale ranging from -3.0 to 
+3.0. 

^Percentage of responses greater than 0.0 on a scale ranging from -3.0 to 
+3.0 
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accuracy of the data. The item "general evaluation design" received an 
average rating of l.ZZ^ which again represents a positive judgment on the 
part of the auditors. The auditor's judgnit/its ranged between 0 and +1 for 
most of the other items (assessment techniques, testing instruments, analysis 
of raw data, appropriateness of data collection techniques, and completeness 
of report. These findings lead to two possible interpretations. One might 
conclude that the auditors were making "cautiously positive*' judgments about 
the evaluators' work in the area of data analysis. On the other hand, the 
auditors may have been content to report that these activities were discussed 
by the evaluator without making any judgments about the quality of the eval- 
uator's work. Whichever was the case, the auditors did not make extreme judg- 
ments in either direction regarding the evaluators' skills in data analysis. 
In general, the auditors viewed the evaluators' work in neutral or only 
slightly positive terms; they appear, on the whole, to have been rather kind 
in their judgments. 

Scope of the Audit Report 

The analysis of the audit report dealt with two issues: the auditor's 
agreement with the evaluator's findings and the areas of the program in which 
modifications were recommended by the auditor. Data on the agreement of the 
auditor with the evaluator's findings were derived from the raters' 
consideration of a number of areas of program outcome such as student learn- 
ing, student attitudinal change, and community /pa rent involvement. In each 
instance, the rater indicated that the auditor (a) agreed with the evaluator^ s 
findings, (b) did not agree with the evaluator's findings, or (c) did not 
comment on evaluation findings with respect to this element of the program. 

ERLC 



8. 



The auditors tended to support the evaluators' conclusions regarding student 
cognitive achievement, staff development, and comniunity/parent involvement. 
Indeed, there were few cases where the auditor specifically disagreed with 
or disputed the evaluator's findings. In those cases where "agreement" was 
not reported, the auditor generally did not comment on the evaluator's find- 
ings. 

Data on auditor's recommendations for program modification were obtained 
by having the rater consider some fifteen areas of program activity, such as 
evaluation design, data analysis techniques, and provision for unanticipated 
outcomes. In each case the rater indicated whether or not the audit report 
contained a recommendation for program modification. The program areas where 
auditors most frequently recommended modifications were the design of the 
evaluation and the data collection techniques. Clearly these areas are 
entirely appropriate for consideration by the auditor. One wishes, in fact, 
that the auditors' recommendations for program modification in these areas 
had been even stronger. The recommendations and conclusions tended to be 
quite vague and in general, were not based upon specific situations or data. 
It is worthwhile to note the areas of the program in which auditors did not 
make recommendations: staff development, community involvement, inservice 
training, and design of the instructional program. Auditors have been warned 
to avoid becoming involved in program management; our data suggest that they 
have followed this warning. 

Summary and Conclusions 

The data on the independtrut euucatlonal accomplishnient audits suggest 
that the auditors performed their services in accordance with the general 
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intent of the USOE guidelines. Those activities in which auditors are expec- 
ted to engage did in fact take place on most occasions. Auditors were able 
to make judgM.ents about evaluation procedures and were able to make recom- 
mendations for modification on those areas of the program related to the 
evaluation. 

The independent program audit is a nev/ concept and, as such, has not 
been implemented with complete perfection. Auditors should be encouraged to 
clearly and precisely detail the activities in v/hich they engaged and which 
serve as the basis for their judgments about the evaluation in particular 
and the program in general. There should be no mystery surrounding the 
procedures which v/ere employed by the auditor. Accountability requires full 
disclosure. The auditors tended to avoid making strong or harsh judgments 
about the evaluation efforts which they reviewed. Their timidity is quite 
understandable. The audits in question were completed some two years ago 
when the concept v/as even newer than it is today. Furthermore, there is an 
inclination among the auditors to practice "professional courtesy" toward 
the evaluator (after all, the roles may be reversed next yearl). 

Finally, this investigation suggests that relatively few auditors have 
attempted to re-analyze an evaluator's data for purposes of verification. 
There are a variety of possible explanations for this fact, including the 
limitations of time and funds and some question as to the values to be derived 
from such activities. It seems that those who are responsible for commission- 
ing audits should give some consideration to the desirability of requiring 
this particular procedure. 

As a postscript to this brief examination of the independent educational 
accomplishment audits it is worthy of note that the audit is no longer 
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required for Title VII projects. Given the importance of the independent audit 
in establishing program accountability and the general success of the Title VII 
audits, this appears to have been a most re'T'^ttable decision. 
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